ISML Machine Learning, Class 12 Lab Kernel Regularized Least Squares

This Lab is about Kernel Regularized Least Squares (KRLS). Follow the instructions below. Think hard before you call the instructors!

Download:

• zip file (unzip it in a local folder)

• Set Matlab path to include the local folder

1. Kernel Regularized Least Squares

• 1.A Generate a 2-class training set by using the following code:

  [Xtr, Ytr] = MixGauss([[0;0],[1;1]],[0.5,0.3],100);

  [Xts, Yts] = MixGauss([[0;0],[1;1]],[0.5,0.3],100);

  Ytr(Ytr==2) = -1;

  Yts(Yts==2) = -1;
  
  

• 1.B Have a look at the code of functions regularizedKernLSTrain and regularizedKernLSTest, and complete them.
  
  

• 1.C Check how the separating function changes with respect to lambda and sigma. Use the Gaussian kernel (kernel='gaussian') and try some regularization parameters, with:
  - sigma in [0.1, 10] ( linspace(0.1 , 10 , 2))
  - and lambda in [0, 10] ([ 0 logspace(-5 , 0 , 2) ])
  To visualize the separating function (and thus get a more general view of what areas are associated with each class) you may use the routine separatingFKernRLS (type "help separatingFKernRLS" on the Matlab shell, if you still have doubts on how to use it, have a look at the code).
  
  

• 1.D Perform the same experiment with noise, by using flipped labels (Ytrn = flipLabels(Ytr, p)) with p equal to 0.05 and 0.1. Check how the separating function changes with respect to lambda.
  
  

• 1.E Load the Two Moons dataset by using the command [Xtr, Ytrn, Xts, Ytsn] = two_moons(npoints, pflipped), where npoints is the number of points in the dataset (between 1 and 100) and pflip is the fraction of flipped labels. Then, visualize the training and the test sets by executing the following instructions:

  figure;

  scatter(Xtr(:,1), Xtr(:,2), 25, Ytrn, 'filled')

  figure;

  scatter(Xts(:,1), Xts(:,2), 25, Ytsn, 'filled')
  
  

• 1.F Perform the exercises 1.C and 1.D on the Two Moons dataset.

2. Parameter Selection

• 2.A By using the dataset in 1.E with 100 points and 0.05 flipped labels, select the suitable lambda by using holdoutCVKernRLS (see help holdoutCVKernRLS for more information), and the sequence

  intKerPar = 0.5;

  intLambda = [5, 2, 1, 0.7, 0.5, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001,0.00001,0.000001];

  nrip = 51;

  perc = 0.5;
  
  

  Then, plot the validation error and the test error with respect to the choice of lambda by running the following code (Note: The x-axis has a logarithmic scale):

  figure;

  semilogx(intLambda, Tm, 'r');

  hold on;

  semilogx(intLambda, Vm, 'b');

  hold off;

  legend('Training','Validation');
  
  

• 2.B Perform the same experiment for different fractions of flipped labels (e.g. p = 0.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5). Check how the training and validation errors change with different p.
  
  

• 2.C Select a suitable sigma by using holdoutCVKernRLS on the following collection (intKerPar), as in exercise 2.A. Then, plot the validation and test error.

  intKerPar = [10, 7, 5, 4, 3, 2.5, 2.0, 1.5, 1.0, 0.7, 0.5, 0.3, 0.2, 0.1, 0.05, 0.03, 0.02, 0.01];

  intLambda = 0.00001;

  nrip = 51;

  perc = 0.5;
  
  

• 2.D Perform the same experiment for different fractions of flipped labels (e.g. p = 0.0, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5). Check how the training and validation errors change with different p.
  
  

• 2.E Now, select the best lambda and sigma by using holdoutCVKernRLS on the following collections, as in exercise 2.A.

  intKerPar = [10, 7, 5, 4, 3, 2.5, 2.0, 1.5, 1.0, 0.7, 0.5, 0.3, 0.2, 0.1, 0.05, 0.03, 0.02, 0.01];

  intLambda = [5, 2, 1, 0.7, 0.5, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001,0.00001,0.000001];

  nrip = 7;

  perc = 0.5;

  Then plot the separating function computed with the best lambda and sigma you have found. (use separatingFKernRLS).
  
  

• 2.F Compute the best lambda and sigma and plot the associated separating functions with 0%, 5%, 10%, 20%, 30%, 40% of flipped labels. How do the parameters differ? And the curves?

3. If you have time - More experiments

• 3.A Repeat the experiment in section 2, with less points (70, 50, 30, 20) and 5% of flipped labels.

  How do the parameters vary with respect to the number of points?
  
  

• 3.B Repeat the experiment in section 1 with the polynomial kernel (kernel = 'polynomial') with parameters lambda in the interval [10, 0] and p, the exponent of the polynomial kernel, in {10,9,...,1}.
  
  

• 3.C Perform Exercise 2.F with the polynomial kernel and the following range of parameters.

  intKerPar = 10:-1:1;

  intLambda = [5, 2, 1, 0.7, 0.5, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001,0.00001,0.000001];

  What is the best exponent for the polynomial kernel on this problem? Why?
  
  

• 3.D Analyze the eigenvalues (see function eig) of the kernel matrix for the polynomial kernel with different values of p (plot them by using semilogy).
  What happens with different p? Why?